Grokking Database Fundamentals for Tech Interviews
Rate this course
Ask Author
Contribute
Back to course home

0% completed

Vote For New Content
Introduction to Database Systems
Course Overview
Overview of Databases
Core Components of Database
Database Architectures
Relational Vs. Non-relational Databases
Data Storage in DBMS
Data Storage Fundamentals
File Organization in DBMS
Data Compression and Encoding
Quiz
Indexing in DBMS
Introduction to Database Indexes
Single Level Indexing
Tree-Based Indexing Techniques
Hash-Based Indexing
Bloom Filters
Quiz
Transaction Management
Understanding the Transaction
Atomicity
Consistency
Isolation
Durability
Quiz
Concurrency Control
Understanding the Concurrency Control
Optimistic vs. Pessimistic Concurrency Control
Lock-Based Concurrency Control
Timestamp-Based Concurrency Control
Multiversion Concurrency Control (MVCC)
Quiz
Distributed Databases, Data Partitioning & Sharding
Fundamentals of Distributed Databases
Partitioning in Databases
Partitioning Types
Sharding in Databases
Distributed Query Processing
Quiz
Replication in Databases
Replication Topologies
Understanding the Consistency Models in Databases
The CAP Theorem and Its Implications
Quiz
Understanding the Consensus
Consensus algorithms
Leader Election Strategies
Quiz
Understanding the Fault Tolerance
Logging Mechanisms in Databases
Checkpointing and Recovery Basics
Coordinated Recovery in Distributed Systems
High Availability Through Replication and Failover
Ensuring Data Resilience: Redundancy, Mirroring, and Recovery
Quiz
Introduction to NoSQL
Key-Value Stores
Document-Oriented Databases
Column-Family and Wide-Column Stores
Graph Databases
Quiz
Quora System Design & Database Design
TikTok System Design & Database Design
Airbnb System Design & Database Design
Snapchat System Design & Database Design
Sharding in Databases

Sharding is a database architecture pattern that splits a single, large database into smaller, more manageable pieces called shards. Each shard is a subset of the database and operates as an independent database. Sharding helps improve performance, scalability, and availability in systems with large amounts of data or high transaction volumes.

How It Works

  • Data is distributed across shards based on a sharding key.
  • Each shard is responsible for a subset of the data.
  • The system routes queries to the appropriate shard(s) based on the sharding key.
Image

Key Characteristics of Sharding

  1. Independent Shards: Each shard operates as a separate database, containing its own data and resources.
  2. Scalability: Shards can be added or removed to handle changing data volumes or traffic patterns.
  3. Distributed Queries: Queries need to be routed to the correct shard(s) based on the sharding key.
  4. No Single Point of Failure: Sharding eliminates the risk of a single point of failure by distributing data across multiple servers.

Example of Sharding

Let’s consider a database of an e-commerce platform that stores customer orders. The Orders table has the following schema:

OrderIDCustomerIDOrderDateAmount
11012023-01-05$50
21022023-02-12$80
31032023-01-18$100
41042023-03-07$40

Using OrderID as the sharding key, the data can be distributed into two shards:

  • Shard 1:

    • Stores data for OrderID values from 1 to 2.
    • Example:
      OrderIDCustomerIDOrderDateAmount
      11012023-01-05$50
      21022023-02-12$80
  • Shard 2:

    • Stores data for OrderID values from 3 to 4.
    • Example:
      OrderIDCustomerIDOrderDateAmount
      31032023-01-18$100
      41042023-03-07$40

When a query is made to retrieve orders for OrderID = 2, the system automatically routes the query to Shard 1 based on the sharding key.

Sharding Key Selection

The choice of a sharding key is critical for the effectiveness of sharding. A good sharding key should:

  1. Distribute Data Evenly: The key should ensure that data is evenly distributed across shards to avoid hotspots or overloaded shards.

  2. Support Query Patterns: The key should align with common query filters, ensuring queries can be routed to specific shards without scanning unnecessary data.

  3. Minimize Rebalancing: The sharding key should reduce the need to move data between shards when scaling or redistributing the database.

For example:

  • A UserID can be an effective sharding key for a social media application to distribute user-specific data across shards.
  • A Timestamp can be a suitable key for a time-series database to shard data by date ranges.

Horizontal Partitioning vs. Sharding

Although sharding is often compared to horizontal partitioning, they differ in implementation and scope. Below is a comparison:

AspectHorizontal PartitioningSharding
DefinitionDividing data into smaller tables or partitions based on rows.Distributing data across multiple databases or nodes.
FocusOrganizing data within a single database.Spreading data across multiple systems in distributed environments.
Data DistributionAll partitions are part of the same database.Shards operate as independent databases.
Query ScopeQueries are processed within the same database.Queries are routed to specific shards based on the sharding key.
Use CaseSuitable for scaling within a single database.Ideal for distributed systems with high transaction volumes.
ImplementationEasier to implement with database-specific features.Requires additional logic for routing and managing shards.
Fault ToleranceRelies on replication within the database.Each shard can have its own replication and failover strategies.

Benefits of Sharding

Once the data is sharded, it brings several benefits:

  • Improved Performance: Queries run faster as they access only a specific shard instead of scanning the entire dataset.
  • Horizontal Scaling: New shards can be added to accommodate growing datasets, ensuring scalability.
  • Fault Isolation: Failures in one shard do not impact the others, improving overall system availability.
  • Efficient Resource Utilization: Workload is distributed across shards, reducing the burden on any single server.

Sharding is an effective database design technique for managing large datasets and scaling horizontally. By splitting data into smaller, distributed pieces, it improves performance, scalability, and availability.

Selecting an appropriate sharding key and understanding the differences between horizontal partitioning and sharding are critical to successfully implementing this strategy. In the next lesson, we will explore Replication in Databases and its importance in distributed systems.

Partitioning Types
Distributed Query Processing
Mark as Completed

On this page

How It Works

Key Characteristics of Sharding

Example of Sharding

Sharding Key Selection

Horizontal Partitioning vs. Sharding

Benefits of Sharding